System and terminal apparatus

ABSTRACT

A system according to the present disclosure includes a first terminal apparatus that repeatedly transmits an image captured using an imager and a second terminal apparatus that displays the received captured image. The first terminal apparatus acquires a position of a display object capable of displaying information, determines, when existence of a blocking object between the display object and the imager is detected, whether a blocked portion, which is a portion where the display object is at least partially blocked by the blocking object, exists within the captured image, and processes, when the blocked portion exists within the captured image, the captured image for the blocked portion within the captured image to be identifiable and transmits a captured image in which the blocked portion is identifiable as the captured image. The second terminal apparatus displays the captured image without displaying the blocked portion based on the received captured image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No.2022-099119, filed on Jun. 20, 2022, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a system and a terminal apparatus.

BACKGROUND

Technology is known for a plurality of computers to communicate via anetwork and hold meetings on the network. In such network meetings,technology is known for capturing information, such as text or picturesdrawn by a user on a display object such as paper or a whiteboard, witha camera and sharing the information. For example, Patent Literature(PTL) 1 discloses a projected image storage system that projects amanuscript onto a whiteboard using a projector, captures a compositeimage of the projected manuscript and the text and numbers written onthe whiteboard with a marker, and stores, displays, and prints thecomposite image.

CITATION LIST Patent Literature

PTL 1: JP 2001-312001 A

SUMMARY

However, there exists demand for further improvement in the usefulnessof the technology for capturing information displayed on a displayobject, such as paper or a whiteboard, and sharing the information overa network. For example, when information written by a user on a displayobject is captured by a camera and shared in real time, the user's handor head may act as a blocking object and end up hiding the informationon the display object, resulting in incorrect sharing (occlusion).

It would be helpful to improve the usefulness of technology forcapturing information displayed on a display object, such as paper or awhiteboard, and sharing the information via a network.

A system according to an embodiment of the present disclosure is asystem including:

-   -   a first terminal apparatus configured to repeatedly transmit a        captured image that is captured using an imager; and    -   a second terminal apparatus configured to display the captured        image received from the first terminal apparatus, wherein    -   the first terminal apparatus is configured to        -   acquire a position of a display object capable of displaying            information,        -   determine, in a case in which existence of a blocking object            between the display object and the imager is detected,            whether a blocked portion, which is a portion where the            display object is at least partially blocked by the blocking            object, exists within the captured image, and        -   process, in a case in which it is determined that the            blocked portion exists within the captured image, the            captured image for the blocked portion within the captured            image to be identifiable and transmit a captured image in            which the blocked portion is identifiable as the captured            image, and    -   the second terminal apparatus is configured to display the        captured image without displaying the blocked portion based on        the captured image received from the first terminal apparatus.

A terminal apparatus according to an embodiment of the presentdisclosure is a terminal apparatus including an imager, a communicationinterface, and a controller and configured to repeatedly transmit acaptured image that is captured using the imager, wherein

-   -   the controller is configured to        -   acquire a position of a display object capable of displaying            information,        -   determine, in a case in which existence of a blocking object            between the display object and the imager is detected,            whether a blocked portion, which is a portion where the            display object is at least partially blocked by the blocking            object, exists within the captured image, and        -   process, in a case in which it is determined that the            blocked portion exists within the captured image, the            captured image for the blocked portion within the captured            image to be identifiable and transmit a captured image in            which the blocked portion is identifiable as the captured            image.

A terminal apparatus according to an embodiment of the presentdisclosure is a terminal apparatus including an output interface, acommunication interface, and a controller and configured to display acaptured image received from another terminal apparatus, wherein

-   -   in a case in which the captured image received from the another        terminal apparatus is a captured image in which a blocked        portion existing within the captured image is identifiable, the        controller is configured to display the captured image without        displaying the blocked portion, and    -   the blocked portion is a portion in the captured image where a        display object configured to display information is at least        partially blocked by a blocking object.

According to an embodiment of the present disclosure, the usefulness oftechnology for capturing information displayed on a display object, suchas paper or a whiteboard, and sharing the information over a network isimproved.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a block diagram illustrating a schematic configuration of asystem according to an embodiment of the present disclosure;

FIG. 2 is a diagram illustrating the positional relationship, fromabove, between a first terminal apparatus illustrated in FIG. 1 and adisplay object;

FIG. 3 is a diagram illustrating the positional relationship, fromabove, between the first terminal apparatus illustrated in FIG. 1 , adisplay object, and a blocking object;

FIG. 4 is a diagram illustrating a captured image captured by the firstterminal apparatus in the positional relationship illustrated in FIG. 2;

FIG. 5 is a diagram illustrating a captured image captured by the firstterminal apparatus in the positional relationship illustrated in FIG. 3;

FIG. 6 is a sequence diagram illustrating example operations of thesystem illustrated in FIG. 1 ;

FIG. 7 is a flowchart illustrating example operations when the terminalapparatus illustrated in FIG. 1 transmits a captured image;

FIG. 8 is a flowchart illustrating example operations when the terminalapparatus illustrated in FIG. 1 displays a received image; and

FIG. 9 is a diagram illustrating a captured image displayed on thesecond terminal apparatus illustrated in FIG. 1 .

DETAILED DESCRIPTION

Hereinafter, an embodiment of the present disclosure will be described.

Outline of Embodiment

An outline of a system 1 according to an embodiment of the presentdisclosure will be described with reference to FIG. 1 . FIG. 1 is ablock diagram illustrating a schematic configuration of the system 1.The system 1 includes a server apparatus 10, a first terminal apparatus20A, and a second terminal apparatus 20B.

The server apparatus 10 is one computer or a plurality of computers thatcan communicate with each other. The first terminal apparatus 20A andsecond terminal apparatus 20B are, for example, personal computers butmay be any computer, such as a smartphone or tablet device. In thepresent embodiment, the first terminal apparatus 20A and the secondterminal apparatus 20B are configured identically. That is, the firstterminal apparatus 20A can also operate as the second terminal apparatus20B, and the second terminal apparatus can also operate as the firstterminal apparatus 20A. Hereinafter, the first terminal apparatus 20Aand the second terminal apparatus 20B are collectively referred tosimply as the terminal apparatus 20 when no distinction therebetween ismade. However, the first terminal apparatus 20A and the second terminalapparatus 20B may be configured differently from each other.

The server apparatus 10, the first terminal apparatus 20A, and thesecond terminal apparatus 20B are communicably connected to a network30. The network 30 may, for example, be the Internet or may include anad hoc network, a local area network (LAN), a metropolitan area network(MAN), other networks, or any combination thereof.

For the sake of simplicity, one server apparatus 10 and two terminalapparatuses 20 are illustrated in FIG. 1 . However, the system 1 mayinclude any number of server apparatuses 10 and terminal apparatuses 20.

First, with reference to FIGS. 1, 2, 3, 4, and 5 , an outline of thepresent embodiment will be described, and details thereof will bedescribed later. FIG. 2 is a diagram illustrating the positionalrelationship, from above, between the first terminal apparatus 20A and adisplay object X. FIG. 3 is a diagram illustrating the positionalrelationship, from above, between the first terminal apparatus 20A, thedisplay object X, and a blocking object Y. FIG. 4 is a diagramillustrating a captured image A captured by the first terminal apparatus20A in the positional relationship illustrated in FIG. 2 . FIG. 5 is adiagram illustrating a captured image B captured by the first terminalapparatus 20A in the positional relationship illustrated in FIG. 3 .

The system 1 according to the present embodiment is a system forproviding online events, for example, in which a plurality of users canparticipate using terminal apparatuses. Online events include webconferencing, videoconferencing, or events in virtual spaces, but theseexamples are not limiting, and online events may be any event conductedby two-way or one-way communication over a network and involving thetransmission and receipt of images.

The system 1 includes the first terminal apparatus 20A configured torepeatedly transmit a captured image that is captured using an imager 22such as a camera and the second terminal apparatus 20B configured todisplay the captured image received from the first terminal apparatus20A. As a result, the captured image A, which is captured by the firstterminal apparatus 20A and has a display object X such as a whiteboardas a subject, as illustrated in FIG. 2 , is displayed on the secondterminal apparatus 20B, as illustrated in FIG. 4 . Subsequently, supposethat the user of the first terminal apparatus 20A writes text on thedisplay object X with a marker. At this time, as illustrated in FIG. 3 ,the user stands between the display object X and the imager 22 of thefirst terminal apparatus 20A, so that the user's body becomes theblocking object Y. A portion of the display object X is consequentlyhidden by the blocking object Y in the captured image B, as illustratedin the form of captured image B in FIG. 5 .

The first terminal apparatus 20A therefore acquires the position of thedisplay object X capable of displaying information. In a case in whichexistence of the blocking object Y between the display object X and theimager 22 of the first terminal apparatus 20A as illustrated in FIG. 3is detected, the first terminal apparatus 20A then determines whether ablocked portion Z, which is a portion where the display object X is atleast partially blocked by the blocking object Y, exists within thecaptured image B. In a case in which it is determined that the blockedportion Z exists within the captured image B, as illustrated in FIG. 5 ,the first terminal apparatus 20A then processes the captured image B forthe blocked portion Z within the captured image B to be identifiable andtransmits a captured image in which the blocked portion Z isidentifiable as the captured image B. This enables the second terminalapparatus 20B to display the captured image B without displaying theblocked portion Z based on the captured image B received from the firstterminal apparatus 20A.

Thus, when the user of the first terminal apparatus 20A captures animage of the display object X, such as a sheet of paper or a whiteboard,and shares the captured image with the user of the second terminalapparatus 20B via the network 30, the display object X may be at leastpartially blocked by the blocking object Y, such as the user's body. Insuch a case, according to the present embodiment, the display object Xcan be displayed on the second terminal apparatus 20B without displayingthe blocking object Y. According to the present embodiment, theusefulness of technology for capturing information displayed on adisplay object, such as paper or a whiteboard, and sharing theinformation over a network is therefore improved.

The display object X is, for example, a sheet of paper or a whiteboard,but these examples are not limiting. The display object X may be anyobject capable of displaying information, such as a screen, a display, atouch panel, or a board. The display object X may have a function fordisplaying information, or it may be capable of displaying informationby being written on by a pen or the like. The blocking object Y is, forexample, a human hand or head, but these examples are not limiting. Theblocking object Y may be any object that can be positioned between thedisplay object X and the first terminal apparatus 20A.

Next, with reference to FIG. 1 , each constituent element of the system1 will be explained in detail.

(Configuration of Server Apparatus)

As illustrated in FIG. 1 , the server apparatus 10 includes acommunication interface 11, an input interface 12, an output interface13, a memory 14, and a controller 15.

The communication interface 11 includes at least one communicationinterface for connecting to the network. The communication interfaceincluded in the communication interface 11 is compliant with, forexample, a mobile communication standard, a wired LAN standard, or awireless LAN standard, but is not limited to these, and may be compliantwith any communication standard. In the present embodiment, the serverapparatus 10 communicates with the terminal apparatus 20 via thecommunication interface 11 and the network 30. The server apparatus 10may communicate with computers other than the terminal apparatus 20according to the present embodiment via the communication interface 11and the network 30.

The input interface 12 includes at least one interface for input. Theinterface for input is, for example, a physical key, a capacitive key, apointing device, a touch screen integrally provided with the display, amicrophone, or the like. The input interface 12 accepts operations toinput information used for operation of the server apparatus 10 andtransmits the inputted information to the controller 15.

The output interface 13 includes at least one interface for output. Theinterface for output is, for example, a display or a speaker. Thedisplay is, for example, a Liquid Crystal Display (LCD) or an organicElectro Luminescent (EL) display. The output interface 13 outputsinformation such as images, sound, or the like. The input interface 12and the output interface 13 may be formed integrally as a touch panel,for example.

The memory 14 includes one or more memories. The memories included inthe memory 14 may each function as, for example, a main memory, anauxiliary memory, or a cache memory. The memory 14 stores anyinformation used for operations of the server apparatus 10. For example,the memory 14 may store a system program, an application program, adatabase, and the like. The information stored in the memory 14 may beupdated with, for example, information acquired from the network 30 viathe communication interface 11.

The controller 15 includes at least one processor, at least oneprogrammable circuit, at least one dedicated circuit, or a combinationof these. The processor is, for example, a general purpose processorsuch as a central processing unit (CPU) or a graphics processing unit(GPU), or a dedicated processor that is dedicated to specificprocessing, but is not limited to these. The programmable circuit is,for example, a field-programmable gate array (FPGA), but is not limitedto this. The dedicated circuit is, for example, an application specificintegrated circuit (ASIC), but is not limited to this. The controller 15controls the operations of the entire server apparatus 10.

(Configuration of Terminal Apparatus)

As illustrated in FIG. 1 , the terminal apparatus 20 includes acommunication interface 21, an imager 22, an input interface 23, anoutput interface 24, a memory 25, and a controller 26.

The communication interface 21 includes at least one communicationinterface for connecting to the network 30. The communication interfaceincluded in the communication interface 21 is compliant with mobilecommunication standards, such as the 4th generation (4G) standard or the5th generation (5G) standard, for example, but is not limited to these,and may be compliant with any communication standard. In the presentembodiment, the terminal apparatus 20 communicates with the serverapparatus 10 and other terminal apparatuses 20 via the communicationinterface 21 and the network 30. The terminal apparatus 20 maycommunicate with computers other than the server apparatus 10 and otherterminal apparatuses 20 according to the present embodiment according tothe present embodiment via the communication interface 21 and thenetwork 30.

The imager 22 includes one or more cameras. In the present embodiment,the camera includes an imaging sensor 22A that converts light intoelectrical signals using an imaging element to generate a captured imageof a subject and a depth sensor 22B that measures the distance to thesubject to generate a depth image. The imaging sensor 22A includes anRGB camera and captures the subject at, for example, 15 to 30 frames persecond to produce a moving image formed by a series of captured images.The depth sensor 22B includes a ToF (Time Of Flight) camera, LiDAR(Light Detection And Ranging), or a stereo camera and generates depthinformation indicating the distance to the subject appearing in acaptured image. The depth information is associated with the capturedimage and is a depth image, for example. However, the depth informationis not limited to depth images and may, for example, be numericalinformation indicating the distance to the subject appearing in thecaptured image. In such cases, the depth information may be added asmetadata to the captured image. The imager 22 transmits the capturedimage and the depth information to the controller 26.

The input interface 23 includes at least one interface for input. Theinterface for input is, for example, a physical key, a capacitive key, apointing device, a touch screen integrally provided with the display, amicrophone, or the like. The input interface 23 accepts operations forinputting information to be used in the operations of the terminalapparatus 20 and transmits the inputted information to the controller26.

The output interface 24 includes at least one interface for output. Theinterface for output is, for example, a display or a speaker. Thedisplay is, for example, a Liquid Crystal Display (LCD) or an organicElectro Luminescent (EL) display. The output interface 24 outputsinformation in the form of images, sound, or the like. The inputinterface 23 and the output interface 24 may be formed integrally as atouch panel, for example.

The memory 25 includes one or more memories. The memories included inthe memory 25 may each function as, for example, a main memory, anauxiliary memory, or a cache memory. The memory 25 stores anyinformation to be used for operations of the terminal apparatus 20. Forexample, the memory 25 may store a system program, an applicationprogram, embedded software, or the like. The information stored in thememory 25 may be updated with, for example, information acquired fromthe network 30 via the communication interface 21.

The controller 26 includes at least one processor, at least oneprogrammable circuit, at least one dedicated circuit, or a combinationof these. The processor is a general purpose processor such as a CPU ora GPU, or a dedicated processor that is dedicated to specificprocessing, for example, but is not limited to these. The programmablecircuit is an FPGA, for example, but is not limited to this. Thededicated circuit is an ASIC, for example, but is not limited to this.The controller 26 controls the operations of the entire terminalapparatus 20.

(Flow of Operations of System)

With reference to FIGS. 6, 7, and 8 , operations of the system 1according to the present embodiment will be described. FIG. 6 is asequence diagram illustrating example operations of the system 1. FIG. 7is a flowchart illustrating example operations when the first terminalapparatus 20A transmits a captured image. FIG. 8 is a flowchartillustrating example operations when the second terminal apparatus 20Bdisplays a received captured image. These operations correspond to amethod performed by the server apparatus 10, the first terminalapparatus 20A, or the second terminal apparatus 20B included in thesystem 1 according to the present embodiment.

In these example operations, it is assumed that the user of the firstterminal apparatus 20A uses the system 1 to capture informationdisplayed on the display object X (for example, text written by the useron a whiteboard) with the imager 22 of the first terminal apparatus 20Aand share the information in real time with the user of the secondterminal apparatus 20B. In these example operations, the display objectX is positioned at a distance d1 from the imager 22 of the firstterminal apparatus 20A, as illustrated in FIG. 2 .

Referring to FIG. 6 , in step S101, the server apparatus 10 initiates anonline event in which the first terminal apparatus 20A and the secondterminal apparatus 20B participate.

Any method can be adopted to initiate the online event. For example, thecontroller 15 of the server apparatus 10 initiates an online event inwhich the terminal apparatus 20A and the terminal apparatus 20Bparticipate, such as a web conference, based on an online eventinitiation request received from the terminal apparatus 20A or thesecond terminal apparatus 20B. In conjunction with the start of theonline event, the controller 15 may communicate with each of the firstterminal apparatus 20A and the second terminal apparatus 20B toauthenticate the users by user ID, password, or the like.

In these example operations, it is assumed that the controller 15 of theserver apparatus 10 conducts the online event by communicating with eachof the first terminal apparatus 20A and the second terminal apparatus20B after the online event starts, transmitting information receivedfrom the first terminal apparatus 20A to the second terminal apparatus20B, and transmitting information received from the second terminalapparatus 20B to the first terminal apparatus 20A. However, after theonline event has started, the first terminal apparatus 20A and thesecond terminal apparatus 20B may conduct the online event bytransmitting information to each other without going through the serverapparatus 10.

After the start of the online event, the captured image yielded bycapturing an image of the display object X is transmitted by the firstterminal apparatus 20A in step S102. Then, in step S103, the capturedimage received from the first terminal apparatus 20A is displayed by thesecond terminal apparatus 20B. By repetition of steps S102 and S103, theimages captured by the first terminal apparatus 20A are displayed as amoving image on the second terminal apparatus 20B.

The operations by which the first terminal apparatus 20A transmits thecaptured image in step S102 are described in detail below with referenceto FIG. 7 .

In step S201, the controller 26 of the first terminal apparatus 20Aacquires the position of the display object X capable of displayinginformation.

Any method can be adopted to acquire the position of the display objectX. For example, the controller 26 of the first terminal apparatus 20Amay store the distance d1 from the imager 22 to the display object X inadvance in the memory 25 as information indicating the position of thedisplay object X. Alternatively, the controller 26 may accept userinput, via the input interface 23, specifying the distance d1 from theimager 22 to the display object X. This enables the controller 26 toidentify the position of the display object X in real space using thedepth sensor 22B of the imager 22.

In step S202, the controller 26 of the first terminal apparatus 20Adetermines whether a blocking object Y exists between the display objectX and the imager 22.

Any appropriate method can be adopted to determine whether the blockingobject exists. For example, the controller 26 of the first terminalapparatus 20A may detect the existence of a blocking object Y betweenthe display object X and the imager 22 using the depth sensor 22B of theimager 22. Specifically, the controller 26 controls the imager 22 togenerate a captured image and a depth image. The controller 26 candetermine that a blocking object Y exists between the display object Xand the imager 22 in a case in which there is a region, in the depthimage generated by the depth sensor 22B, that indicates a distance thatis less than the distance d1 from the imager 22 to the display object X.Conversely, the controller 26 can determine that there is no blockingobject Y between the display object X and the imager 22 in a case inwhich there is no region, in the depth image, indicating a distance lessthan the distance d1.

For example, as illustrated in FIG. 3 , in a case in which thecontroller 26 of the first terminal apparatus 20A detects the existenceof a blocking object Y between the display object X and the imager 22(step S202: YES), the process advances to step S203. In step S203, thecontroller 26 determines whether a blocked portion Z, which is a portionwhere the display object X is at least partially blocked by the blockingobject Y, exists in the captured image.

Any method can be adopted to determine whether the blocked portion Zexists in the captured image. For example, the controller 26 of thefirst terminal apparatus 20A identifies the position of the displayobject X displayed in the captured image and depth image based on theposition of the display object X acquired in step S201. In a case inwhich there is a region in the depth image indicating a distance lessthan the distance d1 (for example, d2) among the positions where thedisplay object X should be displayed, the controller 26 may determinethat the region is the blocked portion Z. The controller 26 can identifythe blocked portion Z in the captured image by mapping the depth imageto the captured image.

For example, as illustrated in FIG. 3 , assume that the user of thefirst terminal apparatus 20A stands at a distance d2 (where d2<d1) fromthe imager 22 of the first terminal apparatus 20A and writes text on thedisplay object X. Consequently, a captured image B as illustrated inFIG. 5 is captured by the first terminal apparatus 20A. In such a case,the controller 26 of the first terminal apparatus 20A detects the user'sbody, located between the display object X and the imager 22 of thefirst terminal apparatus 20A, as the blocking object Y using the depthsensor 22B. The controller 26 can, as a result, identify the shadedportion in FIG. 5 as the blocked portion Z in the captured image B.

Referring again to FIG. 7 , in a case in which the controller 26 of thefirst terminal apparatus 20A determines that the blocked portion Zexists in the captured image B (step S203: YES), the process advances tostep S204. In step S204, the controller 26 processes the captured imageB for the blocked portion Z within the captured image B to beidentifiable and transmits a captured image in which the blocked portionZ is identifiable as the captured image B.

In step S204, the controller 26 of the first terminal apparatus 20A mayuse any method to process the captured image B to be a captured imagesuch that the blocked portion Z within the captured image B isidentifiable. For example, the controller 26 may generate a capturedimage such that the blocked portion Z is identifiable by addingmetadata, to the captured image B, indicating the position of theblocked portion Z in the captured image B (for example, coordinateinformation for the blocked portion Z in the captured image).Alternatively, the controller 26 may generate a captured image in whichthe blocked portion is identifiable by converting the color of theblocked portion Z in the captured image B to a specific color indicatingthe blocked portion Z. The controller 26 transmits the generatedcaptured image in which the blocked portion Z is identifiable to thesecond terminal apparatus 20B as the captured image B. This enables thesecond terminal apparatus 20B to identify the blocked portion Z in thecaptured image B and display the captured image B without displaying theblocked portion Z.

For example, in step S204, the controller 26 may generate, as thecaptured image B in which the blocked portion is identifiable, an imageyielded by deleting pixel information, from the captured image B, forthe region in which the blocked portion Z is displayed. In other words,the captured image in which the blocked portion is identifiable may be acaptured image such that pixel information for the blocked portion Z isdeleted from the captured image B. The pixel information includes, forexample, positional information for pixels and color information forpixels in the captured image. By the first terminal apparatus 20A thusdeleting the pixel information for the blocked portion Z from thecaptured image B in advance and transmitting the result, the amount ofcommunication data associated with the transmission of the capturedimage B can be reduced, and the display processing load for displayingthe captured image B without displaying the blocked portion Z can bereduced on the second terminal apparatus 20B.

Referring again to FIG. 7 , in a case in which the controller 26 of thefirst terminal apparatus 20A determines no blocking object Y existsbetween the display object X and the imager 22 (step S202: NO), or thatno blocked portion Z exists in the captured image (step S203: NO), theprocess advances to step S205. In step S205, the controller 26 transmitsthe captured image that was captured by the imager 22.

When transmitting the captured image, the controller 26 of the firstterminal apparatus 20A may transmit depth information, such as a depthimage, along with the captured image. This makes it possible to performmore advanced display processing, such as three-dimensional display, onthe second terminal apparatus 20B side.

Next, the operations by which the second terminal apparatus 20B displaysthe captured image, received from the first terminal apparatus 20A, instep S103 of FIG. 6 are described in detail with reference to FIG. 8 .For the sake of explanation, it is assumed that the second terminalapparatus 20B has newly received the captured image B illustrated inFIG. 5 while having already received and displayed the captured image Ain FIG. 4 . In the following explanation, the captured image A is alsoreferred to as the captured image A already being displayed, and thecaptured image B is also referred to as the new captured image B.

In step S301, the controller 26 of the second terminal apparatus 20Bdetermines whether the captured image B received from the first terminalapparatus 20A is a captured image in which the blocked portion isidentifiable.

In step S301, the controller 26 of the second terminal apparatus 20B mayuse any method to determine whether the received captured image B is acaptured image in which the blocked portion Z is identifiable. Forexample, in a case in which metadata indicating the position of theblocked portion Z in the captured image B has been added to the capturedimage B, the controller 26 may determine that the captured image B is acaptured image such that the blocked portion Z is identifiable. Asanother example, in a case in which a specific color indicating theblocked portion Z is included in the captured image B, the controller 26may determine that the captured image B is a captured image such thatthe blocked portion Z is identifiable. Alternatively, in a case in whicha portion of the pixel information has been deleted from the receivedcaptured image B, the controller 26 may determine that the capturedimage B is a captured image such that the blocked portion Z isidentifiable.

In a case in which the controller 26 of the second terminal apparatus20B determines that the captured image B received from the firstterminal apparatus 20A is a captured image in which the blocked portionZ is identifiable (step S301: Yes), the process advances to step S302.In step S302, the second terminal apparatus 20B displays the capturedimage B without displaying the blocked portion Z based on the capturedimage B received from the first terminal apparatus 20A.

In step S302, the controller 26 of the second terminal apparatus 20B mayuse any method to display the captured image B without displaying theblocked portion Z. For example, in step S302, when displaying the newcaptured image B received from the first terminal apparatus 20A, thecontroller 26 may display the new captured image B without displayingthe blocked portion Z, in a case in which it is determined that theblocked portion Z exists in the new captured image B, by supplementingthe blocked portion Z in the new captured image B with a portion Z′corresponding to the blocked portion Z in the captured image A that isalready being displayed.

For example, suppose that the second terminal apparatus 20B receives thenew captured image B illustrated in FIG. 5 while displaying the capturedimage A illustrated in FIG. 4 . At this time, the controller 26 of thesecond terminal apparatus 20B identifies the blocked portion Z in thecaptured image B, as illustrated by the dashed line in FIG. 5 . Thecontroller 26 deletes the pixel information for the blocked portion Zfrom the new captured image B and then superimposes the new capturedimage B on the captured image A already being displayed. The controller26 can thereby generate a captured image C in which the blocked portionZ of the new captured image B is supplemented by the portion Z′corresponding to the blocked portion Z in the captured image A alreadybeing displayed, as illustrated in FIG. 9 . FIG. 9 is a diagramillustrating the captured image C displayed on the second terminalapparatus 20B. By displaying the captured image C as the new capturedimage B via the output interface 24 in this way, the controller 26 candisplay the captured image after supplementing the information on thedisplay object X that is hidden by the blocking object Y.

As another example, in step S302, when displaying the new captured imageB received from the first terminal apparatus 20A, the controller 26 ofthe second display apparatus 20B may, in a case in which it isdetermined that the blocked portion Z exists in the new captured imageB, identify an information display portion, in which information isdisplayed, from a portion of the display object X in the new capturedimage B other than the blocked portion Z. The controller 26 may displaythe new captured image B without displaying the blocked portion Z bysuperimposing the information display portion on the display object X inthe captured image A that is already being displayed.

For example, suppose that the second terminal apparatus 20B receives thenew captured image B illustrated in FIG. 5 while displaying the capturedimage A illustrated in FIG. 4 . At this time, the controller 26 of thesecond terminal apparatus 20B identifies the blocked portion Z in thecaptured image B, as illustrated by the dashed line in FIG. 5 . Thecontroller 26 identifies “ABC” and “E”, which appear in a portion otherthan the blocked portion Z in the display object X in the new capturedimage B, as the information display portion. The controller 26superimposes the identified information display portion “ABC” and “E” onthe captured image A already being displayed. This enables thecontroller 26 to display the captured image C as the new captured imageB via the output interface 24, as illustrated in FIG. 9 , following thecaptured image A. In this way, when the new captured image B isdisplayed, the differences in the display object X are sequentiallyreflected in the captured image A that is already being displayed,enabling a real time display of the process of information beingdisplayed on the display object X.

Here, any method can be adopted to identify the information displayportion in which the information for the display object X is displayed.For example, the controller 26 of the second terminal apparatus 20B maystore a first color, which is a base color of the display object X, inthe memory 25 in advance. This enables the controller 26 to identify theportion, of the display object X in the new captured image B, displayedin a different color from the first color as the information displayportion in step S302.

For example, in a case in which the base color of the display object X,which is a whiteboard, is white, the controller 26 of the secondterminal apparatus 20B may identify a non-white portion in the displayobject X as the information display portion. By a portion that is adifferent color from the base color of the display object X beingidentified in this way as information displayed on the display object X,the information displayed on the display object X can be displayedwithout omission.

Alternatively, the second terminal apparatus 20B may store a secondcolor that is the color of the information display portion in advance,and in step S302, may thereby identify a portion, displayed in thedisplay object X in the new captured image B, that is the second coloras the information display portion.

For example, the controller 26 of the second terminal apparatus 20B mayidentify a portion that is the color of the marker, such as black, red,or blue, as the information display portion. By a portion that is aspecific color being identified in this way as information displayed onthe display object X, information other than the information displayedon the display object X, i.e., noise, is less likely to be displayed.

However, in a case in which the information display portion of thedisplay object X is identified, the shadow of the blocking object Y mayappear on the surface of the display object X. Due to being located atthe same depth as the display object X, the shadow portion is notdetermined to be a portion of the blocking object Y despite use of thedepth sensor 22B. Therefore, the shadow of the blocking object Y on thesurface of the display object X may end up being identified as a portionof the information display portion in the above-described processing.

For such a case, in the above-described processing, the controller 26 ofthe second terminal apparatus 20B may further identify the informationdisplay portion in the captured image A already being displayed. Thecontroller 26 may, in a case in which at least a portion of theinformation display portion in the captured image A that is alreadybeing displayed is not identified as either the blocked portion Z or theinformation display portion in the new captured image B, displayinformation in the new captured image B instead of the at least aportion of the information display portion in the captured image A thatis already being displayed. In a case in which the shadow of theblocking object Y temporarily appears on the surface of the displayobject X and is mistakenly recognized as the information displayportion, this prevents the shadow from continuing to be displayed in thecaptured image as part of the information display portion after theshadow disappears from the surface of the display object X.

As described above, the system 1 according to the present embodimentincludes the first terminal apparatus 20A configured to repeatedlytransmit a captured image that is captured using the imager 22 and thesecond terminal apparatus 20B configured to display the captured imagereceived from the first terminal apparatus 20A. The first terminalapparatus 20A acquires the position of the display object X capable ofdisplaying information, and in a case in which existence of the blockingobject Y between the display object X and the imager 22 is detected, thefirst terminal apparatus 20A determines whether the blocked portion Z,which is a portion where the display object X is at least partiallyblocked by the blocking object Y, exists within the captured image. In acase in which it is determined that the blocked portion Z exists withinthe captured image, the first terminal apparatus 20A processes thecaptured image for the blocked portion Z within the captured image to beidentifiable. The first terminal apparatus 20A then transmits a capturedimage in which the blocked portion Z is identifiable as the capturedimage. The second terminal apparatus 20B displays the captured imagewithout displaying the blocked portion Z based on the captured imagereceived from the first terminal apparatus 20A.

When the user of the first terminal apparatus 20A captures an image ofthe display object X, such as a sheet of paper or a whiteboard, thedisplay object X may be at least partially blocked by the blockingobject Y, such as the user's body. Even in this case, however, accordingto the above configuration, the second terminal apparatus 20B can stilldisplay the captured image received from the first terminal apparatus20A without displaying the blocking object Y. According to the presentembodiment, the usefulness of technology for capturing informationdisplayed on a display object, such as paper or a whiteboard, andsharing the information over a network is therefore improved.

While the present disclosure has been described with reference to thedrawings and examples, it should be noted that various modifications andrevisions may be implemented by those skilled in the art based on thepresent disclosure. Accordingly, such modifications and revisions areincluded within the scope of the present disclosure. For example,functions or the like included in each component, each step, or the likecan be rearranged without logical inconsistency, and a plurality ofcomponents, steps, or the like can be combined into one or divided.

For example, an embodiment in which a general purpose computer functionsas the server apparatus 10, the first terminal apparatus 20A, or thesecond terminal apparatus 20B according to the above embodiment can alsobe implemented. Specifically, a program in which processes for realizingthe functions of the server apparatus 10, the first terminal apparatus20A, or the second terminal apparatus 20B according to the aboveembodiment are written may be stored in a memory of a general purposecomputer, and the program may be read and executed by a processor.Accordingly, the present disclosure can also be implemented as a programexecutable by a processor, or a non-transitory computer readable mediumstoring the program.

For example, in the above embodiment, the first terminal apparatus 20Aor the second terminal apparatus 20B may be configured to execute someor all of the operations and processing executed by the server apparatus10. Similarly, the server apparatus 10 may be configured to execute someof the operations and processing executed by the first terminalapparatus 20A or the second terminal apparatus 20B. For example, theserver apparatus 10 may receive the captured images and depth imagescaptured by the first terminal apparatus 20A from the first terminalapparatus 20A, execute each of the processes described as processes ofthe first terminal apparatus 20A and the second terminal apparatus 20B,generate the captured images to be displayed on the second terminalapparatus 20B, and transmit the captured images to the second terminalapparatus 20B. This can reduce the processing load on the terminalapparatus 20 side and enables use of a less expensive computer as theterminal apparatus 20.

Alternatively, the first terminal apparatus 20A may be configured toexecute some of the operations and processing executed by the secondterminal apparatus 20B. After capturing the captured images and depthimages, the first terminal apparatus 20A may execute each of theprocesses described as processes of the second terminal apparatus 20B,generate the captured images to be displayed on the second terminalapparatus 20B, and transmit the captured images to the second terminalapparatus 20B. This can reduce the processing load on the secondterminal apparatus 20B side and enables use of a less expensive computeras the second terminal apparatus 20B.

Examples of some embodiments of the present disclosure are describedbelow. However, it should be noted that the embodiments of the presentdisclosure are not limited to these examples.

[Appendix 1] A system comprising:

-   -   a first terminal apparatus configured to repeatedly transmit a        captured image that is captured using an imager; and    -   a second terminal apparatus configured to display the captured        image received from the first terminal apparatus, wherein    -   the first terminal apparatus is configured to        -   acquire a position of a display object capable of displaying            information,        -   determine, in a case in which existence of a blocking object            between the display object and the imager is detected,            whether a blocked portion, which is a portion where the            display object is at least partially blocked by the blocking            object, exists within the captured image, and        -   process, in a case in which it is determined that the            blocked portion exists within the captured image, the            captured image for the blocked portion within the captured            image to be identifiable and transmit a captured image in            which the blocked portion is identifiable as the captured            image, and    -   the second terminal apparatus is configured to display the        captured image without displaying the blocked portion based on        the captured image received from the first terminal apparatus.        [Appendix 2] The system according to appendix 1, wherein    -   the imager comprises a depth sensor, and    -   the first terminal apparatus is configured to detect the        existence of the blocking object between the display object and        the imager using the depth sensor.        [Appendix 3] The system according to appendix 1 or 2, wherein        the captured image in which the blocked portion is identifiable        is a captured image in which pixel information for the blocked        portion is deleted from the captured image.        [Appendix 4] The system according to any one of appendices 1 to        3, wherein when displaying a new captured image received from        the first terminal apparatus, the second terminal apparatus is        configured to display the new captured image without displaying        the blocked portion, in a case in which it is determined that        the blocked portion exists in the new captured image, by        supplementing the blocked portion in the new captured image with        a portion corresponding to the blocked portion in a captured        image that is already being displayed.        [Appendix 5] The system according to any one of appendices 1 to        3, wherein when displaying a new captured image received from        the first terminal apparatus, the second terminal apparatus is        configured to display the new captured image without displaying        the blocked portion, in a case in which it is determined that        the blocked portion exists in the new captured image, by        identifying an information display portion, in which information        is displayed, from a portion of the display object in the new        captured image other than the blocked portion, and superimposing        the information display portion on the display object in a        captured image that is already being displayed.        [Appendix 6] The system according to appendix 5, wherein the        second terminal apparatus is configured to store in advance a        first color that is a base color of the display object and        identify a portion, displayed in the display object in the new        captured image, that is a different color from the first color        as the information display portion.        [Appendix 7] The system according to appendix 5, wherein the        second terminal apparatus is configured to store in advance a        second color that is a color of the information display portion        and identify a portion, displayed in the display object in the        new captured image, that is the second color as the information        display portion.        [Appendix 8] The system according to any one of appendices 5 to        7, wherein the second terminal apparatus is further configured        to identify an information display portion in the captured image        that is already being displayed, and in a case in which at least        a portion of the information display portion in the captured        image that is already being displayed is not identified as        either the blocked portion or the information display portion in        the new captured image, display information in the new captured        image instead of the at least a portion of the information        display portion in the captured image that is already being        displayed.        [Appendix 9] A terminal apparatus comprising an imager, a        communication interface, and a controller and configured to        repeatedly transmit a captured image that is captured using the        imager, wherein    -   the controller is configured to        -   acquire a position of a display object capable of displaying            information,        -   determine, in a case in which existence of a blocking object            between the display object and the imager is detected,            whether a blocked portion, which is a portion where the            display object is at least partially blocked by the blocking            object, exists within the captured image, and        -   process, in a case in which it is determined that the            blocked portion exists within the captured image, the            captured image for the blocked portion within the captured            image to be identifiable and transmit a captured image in            which the blocked portion is identifiable as the captured            image.            [Appendix 10] The terminal apparatus according to appendix            9, wherein    -   the imager comprises a depth sensor, and    -   the controller is configured to detect the existence of the        blocking object between the display object and the imager using        the depth sensor.        [Appendix 11] The terminal apparatus according to appendix 9 or        10, wherein the captured image in which the blocked portion is        identifiable is a captured image in which pixel information for        the blocked portion is deleted from the captured image.        [Appendix 12] A terminal apparatus comprising an output        interface, a communication interface, and a controller and        configured to display a captured image received from another        terminal apparatus, wherein    -   in a case in which the captured image received from the another        terminal apparatus is a captured image in which a blocked        portion existing within the captured image is identifiable, the        controller is configured to display the captured image without        displaying the blocked portion, and    -   the blocked portion is a portion in the captured image where a        display object configured to display information is at least        partially blocked by a blocking object.        [Appendix 13] The terminal apparatus according to appendix 12,        wherein the captured image in which the blocked portion is        identifiable is a captured image in which pixel information for        the blocked portion is deleted from the captured image.        [Appendix 14] The terminal apparatus according to appendix 12 or        13, wherein when displaying a new captured image received from        the another terminal apparatus, the controller is configured to        display the new captured image without displaying the blocked        portion, in a case in which it is determined that the blocked        portion exists in the new captured image, by supplementing the        blocked portion in the new captured image with a portion        corresponding to the blocked portion in a captured image that is        already being displayed.        [Appendix 15] The terminal apparatus according to appendix 12 or        13, wherein when displaying a new captured image received from        the another terminal apparatus, the controller is configured to        display the new captured image without displaying the blocked        portion, in a case in which it is determined that the blocked        portion exists in the new captured image, by identifying an        information display portion, in which information is displayed,        from a portion of the display object in the new captured image        other than the blocked portion, and superimposing the        information display portion on the display object in a captured        image that is already being displayed.        [Appendix 16] The terminal apparatus according to appendix 15,        wherein the controller is configured to store in advance a first        color that is a base color of the display object and identify a        portion, displayed in the display object in the new captured        image, that is a different color from the first color as the        information display portion.        [Appendix 17] The terminal apparatus according to appendix 15,        wherein the controller is configured to store in advance a        second color that is a color of the information display portion        and identify a portion, displayed in the display object in the        new captured image, that is the second color as the information        display portion.        [Appendix 18] The terminal apparatus according to any one of        appendices 15 to 17, wherein the controller is further        configured to identify an information display portion in the        captured image that is already being displayed, and in a case in        which at least a portion of the information display portion in        the captured image that is already being displayed is not        identified as either the blocked portion or the at least a        portion of the information display portion in the new captured        image, display information in the new captured image instead of        the at least a portion of the information display portion in the        captured image that is already being displayed.

1. A system comprising: a first terminal apparatus configured torepeatedly transmit a captured image that is captured using an imager;and a second terminal apparatus configured to display the captured imagereceived from the first terminal apparatus, wherein the first terminalapparatus is configured to acquire a position of a display objectcapable of displaying information, determine, in a case in whichexistence of a blocking object between the display object and the imageris detected, whether a blocked portion, which is a portion where thedisplay object is at least partially blocked by the blocking object,exists within the captured image, and process, in a case in which it isdetermined that the blocked portion exists within the captured image,the captured image for the blocked portion within the captured image tobe identifiable and transmit a captured image in which the blockedportion is identifiable as the captured image, and the second terminalapparatus is configured to display the captured image without displayingthe blocked portion based on the captured image received from the firstterminal apparatus.
 2. The system according to claim 1, wherein theimager comprises a depth sensor, and the first terminal apparatus isconfigured to detect the existence of the blocking object between thedisplay object and the imager using the depth sensor.
 3. The systemaccording to claim 1, wherein the captured image in which the blockedportion is identifiable is a captured image in which pixel informationfor the blocked portion is deleted from the captured image.
 4. Thesystem according to claim 1, wherein when displaying a new capturedimage received from the first terminal apparatus, the second terminalapparatus is configured to display the new captured image withoutdisplaying the blocked portion, in a case in which it is determined thatthe blocked portion exists in the new captured image, by supplementingthe blocked portion in the new captured image with a portioncorresponding to the blocked portion in a captured image that is alreadybeing displayed.
 5. The system according to claim 1, wherein whendisplaying a new captured image received from the first terminalapparatus, the second terminal apparatus is configured to display thenew captured image without displaying the blocked portion, in a case inwhich it is determined that the blocked portion exists in the newcaptured image, by identifying an information display portion, in whichinformation is displayed, from a portion of the display object in thenew captured image other than the blocked portion, and superimposing theinformation display portion on the display object in a captured imagethat is already being displayed.
 6. The system according to claim 5,wherein the second terminal apparatus is configured to store in advancea first color that is a base color of the display object and identify aportion, displayed in the display object in the new captured image, thatis a different color from the first color as the information displayportion.
 7. The system according to claim 5, wherein the second terminalapparatus is configured to store in advance a second color that is acolor of the information display portion and identify a portion,displayed in the display object in the new captured image, that is thesecond color as the information display portion.
 8. The system accordingto claim 5, wherein the second terminal apparatus is further configuredto identify an information display portion in the captured image that isalready being displayed, and in a case in which at least a portion ofthe information display portion in the captured image that is alreadybeing displayed is not identified as either the blocked portion or theinformation display portion in the new captured image, displayinformation in the new captured image instead of the at least a portionof the information display portion in the captured image that is alreadybeing displayed.
 9. A terminal apparatus comprising an imager, acommunication interface, and a controller and configured to repeatedlytransmit a captured image that is captured using the imager, wherein thecontroller is configured to acquire a position of a display objectcapable of displaying information, determine, in a case in whichexistence of a blocking object between the display object and the imageris detected, whether a blocked portion, which is a portion where thedisplay object is at least partially blocked by the blocking object,exists within the captured image, and process, in a case in which it isdetermined that the blocked portion exists within the captured image,the captured image for the blocked portion within the captured image tobe identifiable and transmit a captured image in which the blockedportion is identifiable as the captured image.
 10. The terminalapparatus according to claim 9, wherein the imager comprises a depthsensor, and the controller is configured to detect the existence of theblocking object between the display object and the imager using thedepth sensor.
 11. The terminal apparatus according to claim 9, whereinthe captured image in which the blocked portion is identifiable is acaptured image in which pixel information for the blocked portion isdeleted from the captured image.
 12. A terminal apparatus comprising anoutput interface, a communication interface, and a controller andconfigured to display a captured image received from another terminalapparatus, wherein in a case in which the captured image received fromthe another terminal apparatus is a captured image in which a blockedportion existing within the captured image is identifiable, thecontroller is configured to display the captured image withoutdisplaying the blocked portion, and the blocked portion is a portion inthe captured image where a display object configured to displayinformation is at least partially blocked by a blocking object.
 13. Theterminal apparatus according to claim 12, wherein the captured image inwhich the blocked portion is identifiable is a captured image in whichpixel information for the blocked portion is deleted from the capturedimage.
 14. The terminal apparatus according to claim 12, wherein whendisplaying a new captured image received from the another terminalapparatus, the controller is configured to display the new capturedimage without displaying the blocked portion, in a case in which it isdetermined that the blocked portion exists in the new captured image, bysupplementing the blocked portion in the new captured image with aportion corresponding to the blocked portion in a captured image that isalready being displayed.
 15. The terminal apparatus according to claim12, wherein when displaying a new captured image received from theanother terminal apparatus, the controller is configured to display thenew captured image without displaying the blocked portion, in a case inwhich it is determined that the blocked portion exists in the newcaptured image, by identifying an information display portion, in whichinformation is displayed, from a portion of the display object in thenew captured image other than the blocked portion, and superimposing theinformation display portion on the display object in a captured imagethat is already being displayed.
 16. The terminal apparatus according toclaim 15, wherein the controller is configured to store in advance afirst color that is a base color of the display object and identify aportion, displayed in the display object in the new captured image, thatis a different color from the first color as the information displayportion.
 17. The terminal apparatus according to claim 15, wherein thecontroller is configured to store in advance a second color that is acolor of the information display portion and identify a portion,displayed in the display object in the new captured image, that is thesecond color as the information display portion.
 18. The terminalapparatus according to claim 15, wherein the controller is furtherconfigured to identify an information display portion in the capturedimage that is already being displayed, and in a case in which at least aportion of the information display portion in the captured image that isalready being displayed is not identified as either the blocked portionor the at least a portion of the information display portion in the newcaptured image, display information in the new captured image instead ofthe at least a portion of the information display portion in thecaptured image that is already being displayed.